Method for registering model data for optical recognition processing and optical sensor

ABSTRACT

To easily generate model data having high recognition accuracy and being consistent with measurement conditions and installation environment of each of optical sensors. Basic model representing a range in which a workpiece can be optically recognized is inputted, and pieces of processing of imaging and measuring the workpiece under the same condition as that in an actual measurement and matching feature data of the workpiece obtained from this measurement with the basic model are executed for a plurality of number of cycles. Then, in the basic model, information is set as unnecessary information where the information cannot be associated with the feature data of the workpiece in all of the pieces of matching processing, or where the number of times or ratio the information cannot be associated is more than a predetermined reference value, or where the information cannot be associated with the feature data in any one of the pieces of executed matching processing. Then, the unnecessary information is deleted from the basic model, and information after each deletion is identified as model data to be registered and is registered to the memory.

This application is based on Japanese Patent Application No. 2009-061355filed with the Japan Patent Office on Mar. 13, 2009, the entire contentof which is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to an optical sensor that executespredetermined recognition processing by imaging an object to berecognized with a camera, executing two-dimensional or three-dimensionalmeasurement processing using the generated image, and matching theobtained feature data with previously registered model data. Moreparticularly, the present invention relates to processing forregistering the model data used in the above recognition processing tothe optical sensor.

2. Related Art

For example, Japanese Patent No. 2961264 discloses a method forgenerating three-dimensional model data of an outline of an object.

In the invention of this Japanese Patent No. 2961264, three-dimensionalinformation is assumed to be reconstructed by stereoscopic measurement,and an actual model of an object is measured multiple times while themeasuring direction is changed in each of the measurement operations.Then, the three-dimensional information reconstructed by each of themeasurement operations is matched with each other and is positioned, sothat model data that can be measured in various directions are generatedby combining the positioned information.

Japanese Patent No. 2961264 and “Stereo Correspondence Using SegmentConnectivity”, Transactions of Information Processing Society of Japan,Vol. 40, No. 8, pages 3219 to 3229, published on August 1999, disclose“segment-based stereo” as a method for reconstructing three-dimensionalinformation of the outline of an object. In this method, edges includedin images constituting a stereoscopic image are divided into segments oflines and curved lines based on connection points and branching points,and stereo-supported search is performed in units of segments, so that apredetermined number of three-dimensional coordinates are calculated.

A process for registering model data to an optical sensor generallyincludes the steps of providing a recognition-target object having ashape preferable for the registration-target optical sensor, actuallyexecuting imaging and measurement, and generating model data based onfeature data obtained from measurement. In addition, experimentalrecognition is performed using the generated model data in order toensure adequate accuracy in recognition, and the model data arecorrected based on the recognition result. If the accuracy is extremelypoor, for example, model data are generated all over again as necessary.As a result, it takes much labor and time to determine model data to beregistered.

As described above, operation for registering models to an opticalsensor is a heavy burden. Accordingly, engineers working at a site witha plurality of production lines executing the same step demand thatmodel data registered to an optical sensor arranged on one of theselines can be exported to an optical sensor arranged on another line.However, in reality, illumination condition may be different dependingon the line, there may be a line that is affected by external light, anduneven characteristics of cameras may result in generating images havingdifferent contrasts. Therefore, the state of an image generated by eachoptical sensor varies, which makes it difficult to register the samemodel data to each of the optical sensors.

On the other hand, in a case where model data are generated for each ofthe optical sensors, and model data are generated by different workersdepending on the sensor, there may be a possibility that the contentsset in the model data may vary depending on the worker, and theunevenness of the model data may result in unevenness in the stabilityof the processing carried out in each of the lines.

With regard to the above issues, the inventors have consideredconverting design data such as CAD data into a data format suitable formeasurement to automatically generate model data that are not affectedby the difference of measurement conditions and installation conditionsof the sensors and importing the thus generated model data into each ofthe optical sensors. However, the model data derived from the designdata include information beyond the measurable range of the opticalsensors. As a result, when the measurement result is matched, the modeldata are found to include a large volume of information that is notassociated with the measurement result, thus reducing the degree ofconsistency instead of increasing it.

SUMMARY

The present invention has been devised to solve the problems describedabove, and aims to easily generate model data having high recognitionaccuracy and being consistent with measurement conditions andinstallation environment of each of optical sensors.

A method for registering model data according to the present inventionis carried out by an optical sensor that executes recognition processingof an object by imaging the object to be recognized with at least onecamera, obtaining feature data representing a shape of the object byperforming measurement processing using the generated image, andmatching the obtained feature data with previously registered modeldata. In the method for registering model data, the following first tofourth steps are executed.

The first step includes inputting, with regard to the object to berecognized, basic model data representing a full-scale shape of theobject in a range in which the object can be optically recognized.

The “range in which the object can be optically recognized” generallymeans the entire range of the surface of the object. However, if thereis any section of the surface of the object that cannot be observed bythe camera of any one of the optical sensors due to a reason that, e.g.,there is a limitation on the attitude of the object during measurement,the section may be excluded from the “range in which the object can beoptically recognized”, so that it is not included in the basic modeldata. However, it is to be understood that the present invention doesnot exclude a possibility of inputting basic model data includinginformation about sections that cannot be optically recognized, such asthe internal structure of the object.

“Three-dimensional information representing a full-scale shape” may beconstituted by information representing the outline shape or may beconstituted by a set of coordinates representing the surface of theobject. In addition, the “three-dimensional information representing afull-scale shape” is not limited to a full-scale representation, andincludes a representation in a size reduced from the full-scale size bya predetermined ratio.

The second step includes performing, for the predetermined number ofcycles, the pieces of processing of: measuring processing and imagingprocessing performed with the camera on an actual model of the object tobe recognized; and processing for matching feature data of the actualmodel obtained by the measurement processing with the basic model dataor data converted from the basic model data (hereinafter referred to as“converted model data”). The third step includes deleting data thatcould not be associated with the feature data of the actual model fromthe data matched with the feature data of the actual model in the secondstep (which means the basic model data or the converted model data thatare processed in the second step), and adopting the remaining data asmodel data to be registered.

According to the above method, the basic model data about the recognizedobject that includes information more than what is needed by theapparatus is inputted, and this basic model data or the converted modeldata generated from the basic model data are matched with themeasurement result of the measurement performed by the apparatus on theactual model. Then, the data that are not associated with the featuredata of the actual model are deleted, as being data unnecessary for theprocessing in the apparatus, from the basic model data or the convertedmodel data that have been subjected to the matching processing.

In the above matching processing, it is preferable that the positionalrelationship between the camera and the actual model is set to be therelationship expected during the actual measurement, and the image isgenerated according to the characteristics of the camera and theenvironment in which the optical sensor is installed. Further, it ispreferable to match the feature data obtained from that image with thebasic model data or the converted data thereof. Therefore, when the datathat are not associated in the matching processing are deleted, theremaining data automatically become suitable for the recognitionprocessing that should be performed by the optical sensor. Consequently,it is possible to eliminate or greatly reduce the operation forverifying the accuracy of the model data. Further, it is no longernecessary for the worker to correct the model data, and therefore,unevenness does not occur in the accuracy of the model data.

Further, since the basic model data is the information representing thefull-scale shape of the object within the range in which the object tobe recognized can be optically recognized, the same basic model data canbe imported into all of the optical sensors that recognize the sameobject, and each of the optical sensors can execute the above method.Therefore, when the basic model data having a sufficient degree ofaccuracy are prepared, each of the optical sensors can make this basicmodel data into model data that is suitable for the recognitionprocessing performed by the apparatus, and can register the model datathus generated.

Regarding the processing of the third step, three types of aspects willbe hereinafter described.

In the third step according to the first aspect, the data that could notbe associated with the feature data of the actual model in the matchingprocessing in all of the cycles executed in the second step aredetermined to be deleted from among the data matched with the featuredata of the actual model.

According to the above aspect, the data corresponding to the featuredata that are hardly measured by the optical sensor carrying out thismethod are determined to be unnecessary information, and the basic modeldata or the converted model data from which the unnecessary informationis removed can be registered as the model data. Therefore, the datacorresponding to the model data can be stably extracted from the featuredata obtained by measurement, and the recognition accuracy can beensured.

In the third step according to the second aspect, the data that couldnot be associated with the feature data of the actual model in thematching processing in the second step for the number of times or ratioequal to or more than a predetermined reference value are determined tobe deleted from among the data matched with the feature data of theactual model.

According to the above aspect, the model data do not include theinformation which can be measured by the optical sensor but of whichmeasurement may become unstable due to variation of illuminationcondition and the like. Therefore, matching processing performed withunstable information can be prevented, and the recognition accuracy canbe improved.

In the third step according to the third aspect, information that couldnot be associated with the feature data of the actual model in any oneof the pieces of matching processing executed in the second step isdetermined to be deleted from among the data matched with the featuredata of the actual model.

According to the above aspect, information that fails to be associatedwith the feature data of the actual model for at least once is deleted,as being unnecessary information, from the basic model data or theconverted model data. Therefore, regarding the object to be recognized,the model data including only the information that can be measuredalmost without fail can be generated. Thus, the stability in recognitionprocessing can be further improved.

In the above method, in a case where three-dimensional measurementprocessing is executed as the measurement processing so as to obtainfeature data representing a three-dimensional shape of an object to berecognized, the first step includes inputting, as basic model data,three-dimensional information representing at least a full-scale shapeof the object within a range in which the object can be opticallyrecognized. The matching processing of the second step includes matchingthe feature data obtained from the actual model with thethree-dimensional information represented by the basic model data.

On the other hand, in the measurement processing, two-dimensionalmeasurement processing may be executed to obtain an edge of an object inan image generated by the camera. In this case, in the first step,two-dimensional information representing a full-scale edge patternappearing in an image that is obtained by imaging the object arranged ina particular attitude with the camera is inputted as the basic modeldata. In the matching processing in the second step, the edge patternobtained from the image of the actual model is matched with an edgepattern represented by the basic model data.

As described above, the method of the present invention can be appliedto both of the apparatus for two-dimensional measurement and theapparatus for three-dimensional measurement.

The optical sensor applied with the above method includes an input unitfor inputting, with regard to the object to be recognized, basic modeldata representing a full-scale shape of the object in anoptically-recognizable range, an actual model processing unit thatperforms, for the predetermined number of cycles, the measuringprocessing and the imaging processing performed with the camera underthe condition that an actual model of the object to be recognized issubjected to the processing and processing for matching feature data ofthe actual model obtained by the measurement processing with the basicmodel data inputted from the input unit or data converted from the basicmodel data, and a model data setting unit for deleting data that couldnot be associated with the feature data of the actual model from thedata matched with the feature data of the actual model in the matchingprocessing executed by the actual model processing unit, and adoptingthe remaining data as model data to be registered.

According to the above configuration, information is previouslygenerated that represents, with a high degree of accuracy, thefull-scale shape of the recognized object within a range in which theobject can be optically recognized. This information is inputted as thebasic model data to the optical sensor. The actual model arrangedaccording to the measurement condition is introduced to the visual fieldof the camera. The pieces of processing performed by the modelprocessing unit and the model setting unit are executed sequentially.Therefore, model data can be generated that are suitable for themeasurement condition and the environment in which the optical sensorsare installed, and the generated model data can be registered.

According to the present invention, for the plurality of optical sensorsfor recognizing the object of the same kind, one piece of basic modeldata is prepared, wherein the one piece of basic model data representsthe full-scale shape of the object recognized by these sensors withinthe range in which the object can be optically recognized. This basicmodel data can be inputted to each of the optical sensors. The basicmodel data can be converted into model data suitable for the measurementcondition, the characteristics of the camera, and the installationenvironment of each of the optical sensors, and the converted model datacan be registered. Therefore, the registration processing of the modeldata can be efficiently carried out by each of the optical sensors.

Further, the actual model of the object to be recognized is imaged andmeasured under the same condition as that in the actual processing, andthe model data are generated according to the information associatedwith this measurement result. Therefore, the accuracy of the model datacan naturally ensured, and stable recognition processing can be executedwith the model data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing a configuration example of a picking system;

FIG. 2 is a block diagram showing an electric configuration of anoptical sensor;

FIG. 3 is a flowchart showing a procedure of a three-dimensionalrecognition processing carried out by the above optical sensor;

FIGS. 4A and 4B are a perspective view and a bottom view, respectively,showing a configuration example of a workpiece to be recognized;

FIGS. 4C, 4D, and 4E are side views each showing a configuration exampleof a workpiece to be recognized;

FIG. 5 is a view showing a data structure of a basic model of theworkpiece shown in FIGS. 4A and 4B;

FIG. 6 is a view showing an arrangement example of a workpiece duringmeasurement;

FIG. 7 is a view showing a data structure of a practical model generatedusing the workpiece in the arrangement of FIG. 6;

FIGS. 8A, 8B, and 8C are views each showing another example of anarrangement of a workpiece during measurement;

FIG. 9 is a view showing a data structure of a practical model generatedusing the workpiece in the arrangement of FIG. 8;

FIG. 10 is a flowchart showing a procedure of model registrationprocessing;

FIGS. 11A, 11B, 11C are views each showing an example in which a datastructure of a practical model changes according to whether a shadow iscast on a workpiece; and

FIG. 12 is a flowchart showing a procedure of model registrationprocessing performed by an optical sensor carrying out two-dimensionalrecognition processing.

DETAILED DESCRIPTION

FIG. 1 shows an example of configuration of a picking system including athree-dimensional optical sensor.

This picking system picks up, one by one, workpieces W contained in acontainer box 6 in a factory, and conveys the workpieces W to apredetermined position. The picking system includes a joint-arm robot 4that performs operation, a robot control apparatus 3 for controlling theoperation of this robot 4, and a three-dimensional optical sensor 100for recognizing the workpieces W to be processed.

A three-dimensional optical sensor 100 includes a stereo camera 1 and arecognition processing apparatus 2. The stereo camera 1 includes threecameras A, B, C arranged side by side. The recognition processingapparatus 2 stores a dedicated program, and is constituted by a personalcomputer connected to a display unit 25 and an input unit 24 such as akeyboard and a mouse.

The recognition processing apparatus 2 obtains three-dimensionalinformation of an outline of a workpiece W by carrying out stereomeasurement processing using the cameras A, B, C. Thereafter, therecognition processing apparatus 2 recognizes the position and theattitude of the workpiece W by matching the obtained three-dimensionalinformation with three-dimensional model data previously registered(hereinafter referred to as a “three-dimensional model”). The robotcontrol apparatus 3 receives the above recognition result transmittedfrom the recognition processing apparatus 2, and controls the operationof the robot 4 based on the received recognition result so that an arm40 of the robot 4 grips the workpiece W.

FIG. 2 is a block diagram showing the configuration of theabove-described three-dimensional optical sensor 100.

As shown in the figure, the recognition processing apparatus 2 includes,e.g., image input units 20A, 20B, 20C respectively corresponding to thecameras A, B, C, a camera drive unit 21, a CPU 22, a memory 23, an inputunit 24, a display unit 25, a communication interface 26, and anexternal disk apparatus 27.

The camera drive unit 21 simultaneously drives the cameras A, B, Caccording to the instructions given by the CPU 22. When calibrationprocessing and processing for registering a three-dimensional model areperformed, the cameras are driven upon an imaging-start instructiongiven by the input unit 24.

During calibration processing and three-dimensional model generationprocessing, the display unit 25 and the input unit 24 are used to inputinformation for setting and give an imaging-start instruction, and areused to display information for supporting operation and allow the userto confirm a projected image of generated three-dimensional model.

The communication interface 26 is used to communicate with the robotcontrol apparatus 3. The external disk drive 27 is used to readinformation from and write information to a storage medium such as acompact disk. In this embodiment, the external disk drive 27 is used toread later-described basic model data.

The memory 23 includes a large capacity memory such as a ROM, a RAM, anda hard disk, and stores programs and setting data for calibrationprocessing, three-dimensional model generation processing, andthree-dimensional recognition processing of the workpiece W. Inaddition, a dedicated area of the memory 23 stores three-dimensionalmodel data and parameters for three-dimensional measurement calculatedby the calibration processing.

The CPU 22 executes the calibration processing based on the programs inthe memory 23, and calculates and registers the parameters forthree-dimensional measurement. Thereafter, the CPU 22 generates modeldata for recognition processing and executes registration processing.Upon executing two kinds of setting processing, the optical sensor 100is ready to execute three-dimensional measurement of the workpiece W andrecognition processing.

FIG. 3 shows processing procedure executed by the three-dimensionaloptical sensor 100 to recognize the workpiece W. The overview ofrecognition processing will be hereinafter described with reference tothis flowchart.

First, the cameras A, B, C execute stereo-imaging (ST1), and extractedges from the generated images (ST2).

Subsequently, the detected edges are thinned (made into data having onepixel width). The thinned edges are divided into segments of lines andcurved lines based on connection points and branching points (ST3, 4).These segments extracted from the edges on a two-dimensional image arehereinafter referred to as “two-dimensional segments”.

Subsequently, processing is executed to associated two-dimensionalsegments related with each other over images (ST5). More specifically,one of the three images is adopted as a reference image, and attentionis paid to two-dimensional segments of this reference image. Atwo-dimensional segment satisfying the following two conditions aresearched from the remaining two images: One conditions is that the notedtwo-dimensional segment satisfies epipolar condition, and the othercondition is that relationship with an adjacent segment matches with thenoted two-dimensional segment. When the two-dimensional segmentsatisfying the conditions in the two images has been found as a resultof this search processing, these are associated with the noted twodimensional segment.

Two-dimensional segments that are not associated between the images areexcluded from the following processing.

When the above associating processing is finished, the process proceedsto ST6, so that processing is executed to reconstruct three-dimensionalinformation for each combination of the associated two-dimensionalsegments. A three-dimensional segment represented by this reconstructedthree-dimensional information will be hereinafter referred to as a“three-dimensional segment”.

Now, a process for reconstructing one three-dimensional segment from oneset of two-dimensional segments associated with each other will bedescribed.

First, two-dimensional segments associated with each other are furtherassociated in units of pixels, and three-dimensional coordinates arecalculated for each set of associated pixels. Further, a line or curvedline approximating a distributed pattern of each three-dimensional imageis set, and the set line or curved line is sampled at a predeterminedinterval. Then, a set of sampled three-dimensional coordinatesassociated with the attribute (line or curved line) according to thedistributed pattern of each three-dimensional coordinate is determinedas a three-dimensional segment.

In ST6, the above processing is executed with respect to all thecombinations of the two-dimensional segments, thus reconstructingthree-dimensional information constituted by a plurality ofthree-dimensional segments representing the outline shape of theworkpiece W. When this processing is finished, the reconstructedthree-dimensional information is matched with the model data previouslyregistered in subsequent ST7, so that the position and the attitude ofthe workpiece W are recognized.

The processing of ST7 described above will be specifically described. Inthis embodiment, three-dimensional segments representing the outlineshapes within the measurable range of workpiece W are registered asmodel data. In ST7, intersecting points of each three-dimensionalsegment are adopted as feature points, and each feature point on themodel data side and each feature point in the reconstructedthree-dimensional information are associated with each other in around-robin manner, so that the degree of consistency between both ofthem is calculated. Then, an association in which the degree ofconsistency is more than a predetermined reference value is determinedto be correct, and a coordinate associated with a representative point(for example barycenter) in the model data is determined to be theposition of the workpiece W. Further, the rotational angle of the modeldata in this correct association is determined to be the attitude of theworkpiece.

When the above matching processing finds a plurality of associations inwhich the degree of consistency with the three-dimensional model is morethan the reference value, the coordinate and the rotational angle aredetermined for each association. Therefore, even where stereomeasurement is carried out on a plurality of workpieces W, each of theworkpieces W can be individually recognized.

When the position and the attitude of the workpiece W are recognized,the recognition result is thereafter outputted to the robot controller 3via the communication interface 26 (ST8), and the processing isterminated.

In order to obtain a certain degree of accuracy in the above recognitionprocessing, it is necessary to register highly accurate model data. Inthis regard, this embodiment is configured as follows. In a computeroutside of the system shown in FIG. 1, CAD data of the workpiece W isconverted into information in a three-dimensional segment format, sothat three-dimensional information representing all of the outline shapeof the workpiece W in actual dimension is generated. The generatedthree-dimensional information is adopted as basic model data(hereinafter referred to as a “basic model”) of the workpiece. Thegenerated basic model is saved into a compact disk and is provided tothe user.

The user loads this compact disk into the external disk drive 27 of therecognition processing apparatus 2, and starts generation processing ofa three-dimensional model. Then, the user arranges the actual model ofthe workpiece W in the same attitude as that in the actual measurement,and placing the actual model of the workpiece W in the measurementregion of the stereo camera 1. Thereupon, the user uses the input unit24 to give an instruction of imaging processing. The recognitionprocessing apparatus 2 executes processing corresponding to steps ST1 toST6 of FIG. 3 in accordance with this instruction, and matches thethree-dimensional information of the reconstructed actual model with thebasic model read from the external disk drive 27.

Further, in this embodiment, the attitude of the workpiece W is changedon every imaging within the range in which the workpiece W can bearranged during measurement, and the above-described stereo measurementand matching processing are performed for a plurality of cycles. Then,information in the three dimensional segments in the basic model sidethat is not associated with the stereo measurement result in any of thecycles of matching processing is determined to be unnecessaryinformation for recognition processing. Then, the basic model from whichthe unnecessary information is deleted is registered to the memory 23 asmodel data used in the recognition processing of FIG. 3 (hereinafterreferred to as a “practical model”).

FIGS. 4A to 4E are a perspective view, a back view (FIGS. 4A and 4B,respectively), and side views seen form three sides (FIGS. 4C, 4D, and4E, respectively) which illustrate a specific exemplary configuration ofthe workpiece W that is to be recognized by the optical sensor. As shownin these figures, the workpiece W according to this embodiment is aplanar body 30 having a thickness, wherein the back surface of theplanar body 30 is integrally arranged with an attachment metal 40 havingthree attachment pieces 41, 42, 43. The attachment pieces 41, 42, 43protrude backward from three sides of the main body section of theattachment metal 40 in such a manner that the attachment pieces 41, 42,43 are inclined with different angles. Further, leading edges of theattachment pieces 41, 42 are bent, whereas the attachment piece 43 isformed in a straightly extending shape. In FIGS. 4A to 4E, referencenumerals 32, 33, 34, 35 denote the surfaces of thickness sections of theplanar body 30.

The above workpiece W is placed such that an upper surface 31 of theplanar body 30 faces upward, so that the planar body 30 is supported inthe horizontal state by the attachment pieces 42, 43 (at this occasion,the attachment piece 41 is floating). Alternatively, as shown in FIG. 8described below, the surface 33 may be bottom, so that the planar body30 is supported in the vertical state.

FIG. 5 is a schematic diagram according to the perspective view of FIG.4 and illustrates a data structure of a basic model of the aboveworkpiece W. This basic model M0 is generated by converting CAD data,and includes three-dimensional segment information (set ofthree-dimensional segments) representing all of the outlines of theworkpiece W (indicated by alternate long and short dashed lines in thefigure).

FIG. 6 is a view showing the actual model of the workpiece W(hereinafter simply referred to as the “workpiece W”) arranged in thesupport state shown in FIG. 4, wherein the workpiece W of FIG. 6 is seenfrom a direction in which the stereo camera 1 measures the workpiece W.However, the attitude of the workpiece W with respect to the measurementdirection is not constant, and there is a possibility that the workpieceW rotates in a direction of arrow f in the figure.

In the example of FIG. 6, measurement is carried out from immediatelyabove the front surface 31 of the planar body 30. Therefore, themeasurement can be performed only on the front surface 31 of the planarbody 30 and near the leading edges of the attachment pieces 41, 42, 43.It is impossible to perform measurement on the back surface 32 of theplanar body 30 and the main body section of the attachment metal 40.

FIG. 7 shows an example of a practical model M1 generated by matchingthe measurement result obtained under the above measurement conditionwith the basic model M0 shown in FIG. 5. In FIG. 7 and FIG. 9 describedlater, deleted information is represented by an extremely thin brokenline.

This practical model M1 is obtained by repeating imaging and measurementupon rotating the workpiece W in a direction of arrow f in FIG. 6 withrespect to the measurement direction by a predetermined angle, matchingthe three-dimensional information obtained by each of the measurementswith the basic model, and deleting information that is not associatedwith the three-dimensional information of the workpiece W from the basicmodel. As a result, the practical model M1 includes only informationcorresponding to actually-measured sections (the front surface of theplanar body 30 and the leading edges of the attachment pieces 41, 42,43).

FIG. 8 is another arrangement example of the workpiece W with respect tothe stereo camera 1. In the example, the workpiece W is arranged suchthat the surface 33 arranged in a direction without any attachment pieceof the planar body 30 is used as a bottom section, so that the planarbody 30 is vertically erect. FIGS. 8A to 8C show three possibleattitudes in which the workpiece W can be arranged with respect to themeasurement direction. In this way, the attitude of the workpiece W isnot constant, but in this example, the back surface of the planar body30 is not to be measured.

FIG. 9 shows an example of a practical model generated by executingmultiple times of stereo measurement on the workpiece W arranged asshown in FIGS. 8A to 8C and matching the three-dimensional informationobtained from each measurement with the basic model M0. This practicalmodel M3 is generated by deleting the information that is not associatedwith the measurement result of the workpiece W from the basic model M0.

As shown in each of the above examples, the practical models MI, M2generated by the optical sensor 100 according to this embodiment isobtained by deleting the information that could not be actually obtainedin the measurement processing on the workpiece W from thethree-dimensional information included in the basic model M0. Any of thepractical models MI, M2 is generated so as to correspond to themeasurement result of the workpiece W arranged in the same attitude asthe attitude of the workpiece W actually recognized. Therefore, when thethree-dimensional information obtained from the measurement processingis correctly associated with the practical model, almost all informationabout the practical model corresponds to the measured three-dimensionalinformation in the recognition processing using the practical models MI,M2. As a result, the workpiece W can be recognized with a sufficientdegree of consistency, and stable recognition processing can beachieved.

FIG. 10 shows a processing procedure when the practical model isgenerated.

This processing starts when the worker loads a compact disk storing thebasic model into the external disk drive 27 and performs readingoperation. First, in the first step (ST11), the basic model is readaccording to the reading operation, and the basic model is stored to awork area of the memory 23.

At this occasion, the worker arranges the workpiece W in the measurementregion of the stereo camera 1 in the same attitude as that in the actualmeasurement, and performs imaging-instruction operation. According tothis operation, ST12 attains “YES”, and the stereo-imaging is executed(ST13). Further, the three-dimensional information of the workpiece W isobtained by performing the measurement processing using the stereo imagegenerated by this imaging (ST14). ST14 of FIG. 10 represents theprocessing corresponding to ST3 to ST6 of FIG. 3. In ST14, a pluralityof three-dimensional segments are obtained as the feature datarepresenting the outline shape of the workpiece W.

In subsequent ST15, the three-dimensional information (feature data)obtained in the above measurement processing is matched with the basicmodel. In this case, the feature point obtained from the measurement andthe feature point on the basic model side are associated with each otherin a round-robin manner in the same manner as the processing of ST7 ofFIG. 3, and processing for obtaining the degree of consistency isrepeated, so that relationship therebetween with the highest degree ofconsistency is identified as correct relationship. In addition, anotherprocessing is also executed, so that the information that is notassociated with the feature data in the above correct relationship isidentified from among the three-dimensional information included in thebasic model.

Thereafter, the worker varies the attitude of the workpiece W within therange expected in the actual measurement, and performsimaging-instruction operation. The optical sensor 100 executes each ofthe pieces of processing of ST13, 14, 15 according to this operation.

Likewise, stereo-imaging, measuring processing, and processing formatching the feature data obtained from the measurement processing withthe basic model are thereafter repeatedly executed according to theimaging-instruction operation. When the worker decides that necessaryimaging has been finished at a predetermined moment and performsterminating operation (“YES” in ST16), the information that is notassociated with the measurement data in any of the matching processingis identified, based on each of the matching processing results, asunnecessary information from among the three-dimensional information inthe basic model (ST17).

Thereafter, the unnecessary information identified is deleted from thebasic model (ST18), and the remaining information is registered as apractical model to the memory 23 (ST19).

The practical model may be registered after the worker makes aconfirmation operation. For example, the information from which theunnecessary information is deleted may be subjected to transparenttransformation to be converted into the coordinate system of the camerasA, B, C, and an projected image generated by this transformationprocessing may be displayed on the display unit 25, so that the workermakes a decision whether the projected image displayed thereon isconsistent with the image of the actual workpiece W.

According to the above procedure, the same basic model M0 is inputted tothe plurality of optical sensors 100 which aim to process the workpieceW, and each of the optical sensors 100 can generate the practical modelaccording to the measurement condition of each apparatus. For example,the basic model M0 having the configuration as shown in FIG. 5 isinputted to the optical sensor 100 that measures the workpiece W underthe condition shown in FIG. 6 and the optical sensor 100 that measuresthe workpiece W under the condition shown in FIG. 8, and the processingof FIG. 10 described above is executed in each sensor 100, so that theformer sensor 100 registers the practical model M1 as shown in FIG. 7and the latter sensor 100 registers the practical model M2 as shown inFIG. 9. As a result, the processing for recognizing the workpiece W canbe stably executed.

Even when the measurement condition of each optical sensor 100 is thesame, and the measurable range varies depending on the difference ofillumination state and camera characteristics, the same basic model maybe inputted to each sensor 100, so that the practical model according toeach measurement accuracy can be generated by the same method as thatshown in FIG. 10.

FIGS. 11A to 11C schematically show the workpiece W constituted by onlythe planar body 30 so as to illustrate the difference of the practicalmodel due to the difference of illumination condition of the workpieceW. In this figure, FIG. 11A shows the basic model of the workpiece W.FIG. 11B shows the workpiece W_(A) on which a shadow is cast. FIG. 11Cshows the workpiece W_(B) on which a shadow is not cast. The practicalmodel M_(A) and the practical model M_(B) are also shown therewith.

In the above example, when the model registration processing isperformed using the workpiece W_(B) on which a shadow is not cast, thepractical model M_(B) includes information of all the outlines to bemeasured. In contrast, when the model registration processing isperformed using the workpiece W_(A) on which a shadow is cast, thepractical model M_(B) does not include information of the outlines onthe surface on which the shadow is cast. As described above, theworkpiece W_(A) measured under the environment in which a shadow is castis set with the practical model that does not include the informationcorresponding to the measurement data that cannot be obtained due to theshadow. Therefore, the recognition processing can be performed with asimilar degree of accuracy to that for the workpiece W_(B) measuredunder the environment in which a shadow is not cast.

When there is a factor, such as effect of external light, that changesthe environment as the time passes, the workpiece W may be imaged andmeasured in each environment. In the matching processing with the basicmodel M0, information is deleted from the basic model M0 in a case wherethe number of times the information is be associated with the featuredata of the workpiece W or a ratio of this number of times with respectto the total number of times of matching is more than a predeterminedreference value, so that the practical model can be generated withoutany information about sections of which measurement is unstable. Thesame thing also applies to a case where there are not only sections ofwhich measurement can be performed but also sections of whichmeasurement cannot be performed.

When higher recognition accuracy is required, information may be deletedas being unnecessary information from the basic model in a case wherethe information cannot be associated with the measurement data in evenone of the plurality of matching processing. With such stricterreference, the practical model can be generated that includes only theinformation about sections of which measurement can be stably performed,so that the recognition processing can be performed more stably.

In the above embodiment, the basic model is assumed to be generated fromthe CAD data, but the method for generating the basic model is notlimited thereto. For example, an actual workpiece W may be placed in ameasurement area of the optical sensor 100 in which preferableenvironment is ensured, and stereo measurement in a plurality ofdirections may be executed on the workpiece W. Thereupon, a basic modelmay be generated by a method for integrating the measurement results(see Japanese Patent No. 2961264), and the basic model may be importedto each of the optical sensors 100.

In the measurement processing in the above embodiment, the processing iscarried out to reconstruct the three-dimensional informationrepresenting the outline shape of the workpiece W. Likewise, the basicmodel and the practical model are also constituted by thethree-dimensional information representing the outline shape. The methodof three-dimensional measurement is not limited thereto.

For example, a set of three-dimensional coordinates representing theshape of the surface of the workpiece W may be obtained by stereomeasurement and light sectioning method, and this set may be matchedwith the registered model data. The three-dimensional coordinate grouprepresenting the entire surface shape of the workpiece W may be inputtedas the basic model to the optical sensor performing the abovemeasurement. Then, the imaging and measurement processing is executed onthe actual workpiece W, and the three-dimensional coordinate group(feature data) obtained by the measurement processing is matched withthe basic model. Thereupon, the basic model from which the coordinatesthat are not associated with the feature data of the workpiece W aredeleted can be registered as the basic model.

Further, the above method for generating the practical model can beapplied to not only the optical sensor for performing three-dimensionalrecognition processing but also an optical sensor for performingtwo-dimensional recognition processing. Hereinafter, processing forgenerating an edge pattern of a model to be registered will bedescribed. In processing, one camera images the workpiece W, and theedge patterns in the generated image are matched with two-dimensionaledge patterns of the registered model. In this processing, a type ofoptical sensor that recognizes the position and the attitude of theworkpiece W will be described as an example.

The hardware configuration of the optical sensor according to thisembodiment is the same as that shown in FIG. 2 except that there is onlyone combination of the cameras and the image input unit. In the belowdescription, the reference numerals of FIG. 2 are used as necessary (thereference numerals of the camera are omitted).

In this embodiment, it is assumed that the optical axis of the camera isfixed to a particular direction (for example, the optical axis isdirected to a vertical direction) and that the workpiece W to beprocessed is arranged such that the workpiece W is always supported by aparticular section (for example, the arrangements shown in FIG. 6 orFIGS. 8A to 8C). In this case, the basic model represents, in actualdimension, the edge patterns of the workpiece W appearing in the image,and is generated by, for example, two-dimensional CAD data. Further, theoptical sensor 100 according to this embodiment identifies, in advance,the magnification rate of the image generated by the camera (actual sizeof one pixel) based on a result of calibration processing, and registersthis magnification rate to the memory 23.

FIG. 12 shows a procedure of processing for generating a practical modelfrom a basic model that is performed by the optical sensor 100configured based on the above assumption. In this processing, first, thebasic model of the edge pattern are read from the compact disk (ST21),and this basic model is converted into a size suitable for the imagebased on the magnification rate registered in the memory 23 (ST22). Theedge pattern having been subjected to this conversion will behereinafter referred to as “converted model”.

The user performs imaging-start instruction operation. Every time theuser performs operation, the user changes the direction of the workpieceW with respect to the measurement direction within the expected rangeaccording to the conditions of the above assumption. Every time thisoperation is performed, the imaging processing of the camera, theprocessing for extracting edges from the generated image, and theprocessing for matching the extracted edge patterns with the convertedmodel are executed (ST23 to ST26).

In the matching processing of ST26, the positional relationship betweenthe edge patterns extracted from the image and the converted model ischanged and is associated on every processing. When the degree ofconsistency becomes the largest, the relationship therebetween isidentified as a correct relationship. Further, the information that isnot associated with the edge to be matched is identified from theconverted model in this correct relationship.

When a termination operation is performed at a predetermined time, theloop of ST23 to ST27 is terminated, and the process proceeds to ST28. Inthis ST28, for the purpose of deleting information of which measurementis unstable, information is identified as unnecessary information in acase where the ratio of information that is not associated with theimage-side edge with respect to the edge information constituting theconverted model exceeds a reference value. Then, this unnecessaryinformation is deleted from the converted model (ST29), and theconverted model from which the unnecessary information is deleted isregistered as a practical model to the memory 23 (ST30).

As described above, when the recognition processing is performed usingthe feature data representing the two-dimensional shape of the workpieceW, the two-dimensional pattern representing the full-scale shape of theworkpiece W within the range in which the camera can optically recognizethe workpiece W is inputted as the basic model to the optical sensor100. Then, the measurement result of the workpiece W arranged in thesame attitude as that in the actual measurement is matched with thebasic model, and the information that is less likely to correspond tothe measurement result is deleted from the basic model. Therefore, thepractical model without any unstable element such as variation ofillumination can be generated. Further, the same basic model may beinputted to the plurality of optical sensors having the same settingcontent about the arrangement of the workpiece W and the opticaldirection of the camera, so that the practical models suitable for eachof the sensors can be generated.

The basic model imported to the optical sensor 100 performing the abovetwo-dimensional recognition processing is not limited to CAD dataeither. For example, with the optical sensor 100 installed in apreferable environment, the processing may be performed to image theworkpiece W and extract edges form the generated image, so that modeldata represented in the magnification rate of this sensor is generated.Thereafter, the model data may be converted into informationrepresenting actual dimension based on the above magnification rate, andthe thus converted model data may be adopted as basic model.Alternatively, a combination of the magnification rate and the modeldata generated with the optical sensor may be adopted as a basic model.

Further, when the same workpiece W is to be processed, but a differentsurface of the workpiece W is treated as the bottom surface depending onthe optical sensor 100 (for example, there are the sensor arranged asshown in the example of FIG. 6 and the sensor arranged as shown in theexample of FIG. 8), the three-dimensional information representing theentire outline shape of the surface of the workpiece W may be inputtedas the basic model to each of the sensors, and the practical models aregenerated. In simple terms, in this case, the three-dimensional basicmodel is inputted to the optical sensor 100, and thereafter, the basicmodel is subjected to rotational correction according to the arrangementstate of the workpiece W for the sensor 100. Further, the correctedbasic model is subjected to transparent transformation to be convertedinto the camera coordinate system. Then, the two-dimensional edgepattern generated by this conversion is adopted as a converted model,and the processing of ST23 and subsequent steps in FIG. 12 are executed.

Lastly, the optical sensor 100 according to the above variousconfigurations can execute the processing for recognizing the positionand the attitude of the workpiece W to be recognized, and in addition,the optical sensor 100 can also be used for the purpose of determiningwhether or not the attitude or the shape of the workpiece W isappropriate based on the degree of consistency in matching of themeasurement result with the model data. On the other hand, the opticalsensor 100 adapted to extract and match two-dimensional edge patternscan recognize not only the position and the attitude of the workpiece Wbut also the height of the workpiece W by showing the model dataregistered according to the method of FIG. 12 in a plurality ofmagnification rates and matching the edge patterns extracted from theimage of the workpiece W with the model data in various magnificationrates.

1. A method for registering model data for optical recognitionprocessing, wherein an optical sensor executes recognition processing ofan object by imaging the object to be recognized with at least onecamera, obtaining feature data representing a shape of the object byperforming measurement processing using the generated image, andmatching the obtained feature data with previously registered modeldata, and wherein the method for registering model data is a method forregistering the model data used by the matching processing, the modeldata registration method comprising: a first step for inputting, withregard to the object to be recognized, basic model data representing afull-scale shape of the object in an optically-recognizable range; asecond step for performing, for the predetermined number of cycles, thepieces of processing of: measuring processing and imaging processingperformed with the camera on an actual model of the object to berecognized; and processing for collating feature data of the actualmodel obtained by the measurement processing with the basic model dataor data converted from the basic model data; and a third step fordeleting data that could not be associated with the feature data of theactual model from the data collated with the feature data of the actualmodel in the second step, and adopting the remaining data as model datato be registered.
 2. The method for registering model data for opticalrecognition processing according to claim 1, wherein in the third step,the data that could not be associated with the feature data of theactual model in the collating processing in all of the cycles executedin the second step are determined to be deleted from among the datacollated with the feature data of the actual model.
 3. The method forregistering model data for optical recognition processing according toclaim 1, wherein in the third step, the data that could not beassociated with the feature data of the actual model for the number oftimes or ratio equal to or less than a predetermined reference value inthe collating processing in the second step are determined to be deletedfrom among the data collated with the feature data of the actual model.4. The method for registering model data for optical recognitionprocessing according to claim 1, wherein in the third step, informationthat could not be associated with the feature data of the actual modelin any one of these collating processing executed in the second step isdetermined to be deleted from among the data collated with the featuredata of the actual model.
 5. The method for registering model data foroptical recognition processing according to claim 1, wherein as themeasurement processing, three-dimensional measurement processing isexecuted to obtain feature data representing a three-dimensional shapeof the object to be recognized, and wherein in the first step,three-dimensional information representing a full-scale shape of theobject that can be optically recognized is inputted as the basic modeldata, and in the collating processing in the second step, the featuredata obtained from the actual model are collated with thethree-dimensional information represented by the basic model data. 6.The method for registering model data for optical recognition processingaccording to claim 1, wherein processing for obtaining an edge in theimage generated by the camera is executed as the measurement processing,and wherein in the first step, two-dimensional information representinga full-scale edge pattern appearing in an image that is obtained by thecamera by imaging the object arranged in a particular attitude isinputted as the basic model data, and in the collating processing in thesecond step, the edge pattern obtained from the image of the actualmodel is collated with an edge pattern represented by the basic modeldata.
 7. An optical sensor for executing recognition processing of anobject by imaging the object to be recognized with at least one camera,obtaining feature data representing a shape of the object by performingmeasurement processing using the generated image, and collating theobtained feature data with previously registered model data, and whereinthe model data registration method is a method for registering the modeldata used by the matching processing, the optical sensor comprising: aninput unit for inputting, with regard to the object to be recognized,basic model data representing a full-scale shape of the object in anoptically-recognizable range; an actual model processing unit thatperforms, for the predetermined number of cycles, the pieces ofprocessing of: measuring processing and imaging processing performedwith the camera under the condition that an actual model of the objectto be recognized is subjected to processing; and processing for matchingfeature data of the actual model obtained by the measurement processingwith the basic model data inputted from the input unit or data convertedfrom the basic model data; and a model data setting unit for deletingdata that could not be associated with the feature data of the actualmodel from the data matched with the feature data of the actual model inthe matching processing executed by the actual model processing unit,and adopting the remaining data as model data to be registered.